Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition

Knowledge distillation, which is a process of transferring complex knowledge learned by heavy network, i.e., teacher, to lightweight student, has emerged as an effective technique for compressing neural networks. To reduce the necessity training large teacher this paper leverages recent self-knowledge distillation approach train student network progressively distilling its own without pre-trained network. Far from existing methods, mainly focus on still images, our proposed Teaching Yourself that targets at videos human action recognition. Our not only designed but also high generalization capability model. In approach, able update itself using best past model, termed preceding then utilized guide present Inspired consistency in state-of-the-art semi-supervised learning we introduce augmentation strategy increase data diversity and improve consistent predictions approach. benchmark been conducted both 3D Resnet-18 ResNet-50 backbone networks evaluated various standard datasets such UCF101, HMDB51, Kinetics400 datasets. The experimental results have shown teaching yourself method significantly improves recognition performance terms accuracy compared supervised methods. We expensive ablation study demonstrate mitigates overconfident dark generates more input variations same point. code available https://github.com/vdquang1991/Self-KD.